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LONG-TERM  GOALS 

The  long-goal  of  this  project  is  to  define,  develop  and  implement  consistent  quantitative  skill  metrics 
to  assess  the  advancements  in  the  Earth  System  Prediction  Capability  (ESPC).  This  work  will  also 
implement  automated  monitoring  systems  for  gauging  performance  of  the  ESPC.  This  is  an  integral 
part  of  the  overall  ESPC  goal  of  developing  a  global  coupled  system  between  the  presently  separate 
atmosphere,  ocean,  surface  wave,  ice  and  land  forecast  systems  for  the  battlespace  environment. 

OBJECTIVES 

Evaluation  and  validation  of  the  ESPC  coupled  model  system  on  the  subseasonal  to  seasonal  time 
scales.  Development  of  metrics  and  process  oriented  diagnostics  for  these  time  scales  and  the 
formulation  and  implementation  of  the  new  score  card  for  the  coupled  system. 

APPROACH 

The  focus  of  this  component  of  ESPC  is  on  development  and  testing  of  the  new  score  card  for  the 
coupled  system  and  a  “library”  of  the  appropriate  process  diagnostics  that  would  allow  us  to  examine 
and  improve  the  coupled  system  and  its  component.  The  existing  metrics  and  diagnostics  for  various 
timescales,  for  deterministic  and  probabilistic  forecasts  are  gathered  and  tested  using  coupled  and 
uncoupled  system  experiments.  Using  this  pool  of  metrics,  we  will  define  and  iterate  on  a  score  card. 
The  new  score  card  will  be  proposed  at  the  end  of  the  first  quarter  of  FY16  and  it  will  be  applied  to  the 
model  hindcasts  as  they  becoming  available.  The  components  of  diagnostic  software  are  also 
transferred  to  other  work  units  such  that  they  can  be  tested  using  the  ensemble  forecasts  from 
international  projects  such  as  S2S,  YOTC/MJO  and  others. 
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WORK  COMPLETED  FOR  FY15 

Work  completed  this  year  focused  on  intraseasonal  time  scale  metrics  and  diagnostics  for  the 
atmosphere  and  the  ocean. 

1)  We  completed  the  first  draft  of  the  white  paper  on  intraseasonal  diagnostics. 

2)  We  researched  metrics  and  process  oriented  diagnostics  used  by  various  operational  centers  for 
intraseasonal  forecasts. 

3)  We  evaluated  metrics  used  for  longer  time  scale  for  example  in  CMIP  experiments  to 
understand  which  of  these  metrics  can  be  used  for  intraseasonal  forecasts. 

4)  We  wrote  the  diagnostic  software  for  a  number  of  these  metrics  and  applied  to  the  coupled 
system  experiments. 

5)  We  developed  and  tested  the  first  version  of  an  ocean  score  card. 

RESULTS 

Tropical  atmospheric  metrics 

The  intraseasonal  tropical  metrics  are  usually  used  to  measure  the  forecast’s  performance  of  MJO, 
monsoons  and  tropical  cyclones.  The  most  widely  accepted  metric  for  MJO  is  the  RMM  index 
(Wheeler  and  Hendon  2004)  based  on  multivariable  EOF  of  meridionally  averaged  zonal  wind  and 
OLR.  The  biggest  challenge  in  applying  this  metric  to  the  model  forecast  is  defining  the  climatology 
and  averages  used  to  filter  out  interannual  variability,  especially  for  OLR  which  strongly  depends  on 
model  formulation.  For  the  models  used  in  the  YOTC/MJO  project  all  climatologies  and  monthly 
averages  used  to  calculate  anomalies  were  based  on  NCEP  reanalysis.  At  present  the  same  approach  is 
used  in  evaluation  of  RMMI  for  NRL  coupled  system,  but  in  the  future  using  the  model  climatology 
will  be  more  beneficial. 


Figure  1.  Monsoon  MH  index,  which  measures  the  strength  of  local  Hadley  circulation  (Flatau  et  al, 
2001)  for  NA  VGEM  analysis  (black)  and  coupled  model  experimental  5  month  forecast.  MH>0  indicates 
the  monsoon  onset.  In  the  observations  the  development  of  convective  events  in  the  Bay  of  Bengal  in 
May  causes  the  increase  of  the  index  and  likely  leads  to  delay  of  the  “proper”  monsoon  as  described  in 
Flatau  et  al  (2001).  In  the  coupled  model  the  early  May  MH  index  increase  is  underestimated  and 
monsoon  onset  can  be  observed  in  late  May  -a  few  days  ahead  of  climatological  onset. 
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Monsoon  variability  can  be  evaluated  using  the  indices  involving  all  India  preeipitation,  zonal  shear  of 
the  zonal  wind,  vertieal  shear  of  the  meridional  wind  (Fig.  1),  kinetic  energy  (Flatau  at  ah,  2001)  and 
EOF  pattern  of  wind  and  OLR  anomalies  of  BSISO  (Boreal  Summer  Intraseasonal  Oscillation)  (Lee  at 
al  2013). 

The  BSISO  index  is  somewhat  analogieal  to  RMMI  index  for  MJO,  but  the  BSISO  EOFS  are  based  on 
two  dimensional  patterns  of  zonal  wind  and  OLR  anomalies  for  the  monsoon  region.  The  projeetion  of 
foreeasted  or  analyzed  anomalies  on  these  EOFs  can  be  used  to  ereate  phase  diagrams  similar  to  these 
used  for  RMMI  (Fig.2).  We  are  presently  working  on  developing  this  methodology  for  use  in  the  NRL 
system. 


The  Canonical  Northward  Propagating  BSISO  Corrpcinent 
(a)  EOF1  (7,2%)  (h)  E0F2  (4.5?S) 


e  July  24.  2011  (StorUr>g  Dot*} 
O  Jun  17,  2008  (Storting  Doto) 


Figure  2.  EOFs  and  a  phase  diagram  for  BSISO  1  from  Lee  at  al.  (2013).  The  EOFs  are  based  on 
OLR  and  850mb  zonal  wind  with  the  meridional  wind  component  obtained  from  regression.  The 
phase  diagram  describes  the  northeastward  and  then  westward  propagation  of  monsoon 

convectively  active  phase. 


Extratropical  metrics 

The  metries  for  the  annular  modes  for  the  northern  and  southern  hemisphere  were  developed  and 
applied  to  NAVGEM  analyses  and  foreeasts.  The  AO  and  AAO  indices  are  based  on  projection  of 
lOOOmb  (for  AO)  and  700mb  (for  AAO)  geopotential  anomalies  on  the  normal  modes  calculated  from 
the  monthly  averaged  fields  of  lOOmb  and  700mb  geopotential  height.  Figure  3  shows  the  annular 
modes  for  November  2011,  thirty  day  forecast  with  the  coupled  system.  The  figure  indicates  that  while 
AO  index  is  reasonably  simulated,  the  Antaretie  index  remains  much  lower  than  its  value  based  on 
either  NCEP  or  NAVGEM  analysis.  That  means  that  the  southern  hemisphere  westerlies  whieh 
strongly  influence  the  atmosphere  ocean  interaetion  may  be  underestimated.  The  problems  with 
southern  hemisphere  winds  in  elimate  simulations  were  recently  highlighted  by  the  CLIVAR  Southern 
Oeean  Working  group  that  suggested  using  the  strength  and  position  of  the  southern  hemispherie 
westerlies  as  one  of  the  important  metrics.  We  plan  to  inelude  this  metric  for  the  NRL  system. 

Fig.  3e  shows  the  AO  indiees  for  various  versions  of  the  system  physics.  The  large  spread  beyond  14 
days  is  consistent  with  the  CPC  analyses  of  GFS  AO  which  indicates  the  laek  of  skilled  foreeasts 
beyond  10  days.  Lin  et  al.  (2005,  2009)  indicates  that  the  skill  of  polar  mode  foreeasts  depends  on 
MJO  amplitude  -  further  analysis  with  the  eoupled  system  forecasts  and  ensemble  foreeasts  from 
ESPC  WU  8  should  determine  if  this  is  true  for  our  system  as  well.  Interestingly  the  uncoupled  version 
of  the  model  produces  the  worst  AO  representation  in  Fig.  3c. 
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Figure  3.  AAO  (a)  and  AO  (b  and  c)  indices  front  NCEP  (solid  black  lines),  NA  VEM  analysis 
(dashed  black  lines),  and  various  50(a,  b)  and  30  (c)  day  forecasts.  The  red  dashed  line  in  Fig  3c 
indicates  the  uncoupled  version  of  the  experiment  shown  in  the  red  solid  line. 

Another  index  used  frequently  in  evaluation  of  the  extratropical  forecast  is  North  Atlantic  Oscillation 
index  (NAO).  The  NAO  index  is  strongly  correlated  with  AO,  suggesting  that  the  NAO  is  not  an 
independent  mode  of  variability  (Jia,  2013).  Unlike  the  AO  index,  the  NAO  index  does  not  have  a 
unique  definition.  It  can  be  defined  either  based  on  the  pressure  difference  between  Lisbon,  Portugal, 
and  Stykkisholmur,  Iceland  (Hurrell  1995),  by  global  patterns  of  surface  pressure  in  the  North  Atlantic 
region  or  by  500mb  geopotential  patterns  in  northern  hemisphere  For  example,  in  Hurrell(2003)  the 
NAO  index  is  defined  as  the  first  EOF  of  the  SLP  over  the  North  Atlantic(  20-80N  90W  40E);  Lin 
(2009)  defines  the  AO  loading  pattern  as  a  second  rotated  EOF  of  the  monthly  means  of  the  500mb 
heights  for  20N-90N,  with  all  seasons  included.  The  CDC  procedure  uses  the  method  described  in 
Bamston  and  Livzey  (1987)  in  which  rotated  EOFs  of  500mb  height  are  calculated  for  each  month  to 
account  for  seasonality.  Figure  4  shows  the  NAO  index  from  CPC  (based  on  an  EOF  procedure)  and 
Lisbon/  Stykkisholmur  pressure  difference  from  NAGEM  analysis.  It  can  be  seen  that  the  indices 
calculated  by  this  two  methods  are  strongly  correlated  and  the  variability  in  the  NAO  is  similar  to  the 
variability  of  the  AO  shown  in  Fig.  3. 
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Figure  4.  The  NAO  index  for  the  same  time  period  as  the  AO  and  AAO  shown  in  Fig.  3  a  and  b.  The 
black  line  indicates  the  index  from  CPC  while  the  red  line  indicates  the  normalized  NAO  2  point 

index  from  NAVGEM  pressure  analyses. 

The  blocking  index  that  reflects  the  meridional  gradient  of  500mb  height  (Tibaldi  and  Molteni,  1990) 
calculated  for  the  November  2011  coupled  forecast  and  analysis  is  shown  in  Fig.  5.  The  figure 
indicates  that  the  forecasts  shows  development  of  the  Eastern  Pacific  blocking  mid-month  while 
Atlantic  blocking  appears  to  be  delayed  compared  to  the  analysis. 


Figure  5.  The  blocking  index  for  the  NAVGEM  analyses  (right)  and  30  day  forecast  (left)  for 
November  1-30  2011.  The  shading  indicates  the  northern  meridional  pressure  gradient  GHGN<10, 
while  the  contours  indicate  southern  meridional  pressure  gradient  GHGS>0  for  each  longitude. 
For  blocking  to  occur  at  a  given  longitude  both  conditions  need  to  be  satisfied. 

An  ocean/ice/waves  scorecard  for  ESPC  coupled  ocean  atmosphere  forecast  performance 

This  section  describes  progress  in  developing  a  scorecard  system  to  monitor  the  ice/ocean/wave 
performance  of  the  HYCOM-NAVGEM  coupled  Earth  System  Prediction  Capability  (ESPC)  targeted 
for  a  2018  Initial  Operational  Capability  (IOC)  at  the  Navy  DoD  Supercomputing  Resource  Center 
(DSRC). 

Since  the  ESPC  scorecard  system  will  run  daily  and  monitor  an  operational  system,  it  needs  to  ingest 
dependable,  regularly  available  data  sources  for  evaluation.  To  accomplish  this  goal  it  is  being 
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designed  to  ingest  data  that  is  available  in  the  operational  data  streams  at  NAVO.  The  data  streams 
currently  available  are  sea  surface  height,  sea  surface  temperature,  profdes  of  temperature  and  salinity 
and  ice  concentration.  There  are  additional  data  types  that  are  desirable  to  add  to  the  data  regularly 
available  such  as  wave  direction  and  period.  We  will  work  with  NAVO  to  secure  these  new  data 
streams. 

Matchups 

The  first  and  most  expensive  step  of  this  metrics  system  is  model  matchups/data  extraction.  The 
archive  files  from  HYCOM  are  large  and  an  efficient  way  to  matchup  the  model  and  observations  is 
crucial.  This  step  will  utilize  the  AutoMetrics  system  (Dykes,  2011)  that  is  currently  operational  at 
NAVO.  This  set  of  software  modules  was  developed  to  produce  model-observation  or  model-model 
comparison  matchups  and  statistics  to  help  assess  ocean  model  performance.  Utilizing  this  existing 
infrastructure  would  avoid  the  creation  of  a  second  matchup  system  to  support/maintain,  and  it  would 
upgrade  a  present  capability  at  NAVO.  The  matchups  are  currently  being  processed  using  the  OCNQC 
(Cummings  and  Smedstad,  2013)  processed  observations  for  temperature  and  salinity,  computing  “on 
the  fly”  sound  speed,  sonic  layer  depth  (SLD)  and  below  layer  gradient  (BLG)  based  on  the  matched 
profdes  to  produce  additional  matches.  Statistics  are  computed  on  location  and  the  matchup  fdes  are 
stored  for  later  access. 

The  capabilities  of  AutoMetrics  are  presently  focused  on  acoustics  metrics,  and  they  will  need  to  be 
upgraded  to  include  wave  and  ice  metrics.  Necessary  upgrades  have  been  identified  and  will  be 
completed  during  FY16. 

Scorecard  Metrics 

In  FY15  we  completed  the  development  of  ocean  scorecard  version  zero  (SVZ)  (Zamudio  et  al.  2015). 
SVZ  has  been  used  to  evaluate  the  performance  of  the  ESPC  coupled  system  and  the  operational 
HYCOM  nowcast/forecast  system.  The  model  evaluations  concentrate  on  the  comparison  of  model 
output  versus  observed  temperature  and  salinity  profiles,  upper  ocean  currents,  and  sea  surface  height. 
The  observed  and  modeled  profiles  were  used  to  calculate  mean  error  (bias),  root  mean  square  error, 
mixed  layer  depth  (MLD),  and  acoustical  proxies  as  SLD,  BLG,  and  cutoff  frequency.  In  addition,  the 
upper  ocean  currents  (which  were  represented  by  drifting  buoys)  were  used  to  calculate  distance  in  km 
(between  the  observed  and  simulated  buoys)  as  a  measure  of  the  degree  to  which  the  observed  and 
modeled  currents  differ  as  a  function  of  time.  An  example  of  the  statistics  produced  by  SVZ  is  shown 
in  Figure  6. 
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GLOBAL 


Figure  6.  The  upper  panel  tracks  the  global  temperature  profile  scores  from  Oct  2014  —  Jan  2015 
for  a  coupled  HYCOM-NA  VGEM forecast  and  the  operational  HYCOM  nowcast/forecast  system 
(GOES  3.0).  As  part  of  the  scorecard  calculations  RMSE  statistics  are  computed  at  each  observation 
location,  which  are  used  to  form  error  maps  (lower  panel).  The  maps  add  spatial  information  about 
the  performance  of  the  models  and  complement  the  scores  produced  by  the  SVZ. 

This  scorecard  framework  in  FY16  (a)  will  be  expanded  to  include  wave  and  iee  metrics  and  (b)  ported 
to  the  NAVO  DSRC  in  preparation  for  monitoring  the  eoupled  system.  We  are  having  discussions  with 
other  work  units  to  (a)  define  the  iee  and  wave  scoreeard  constituents  and  (b)  identify  the  observation 
data  streams  needed  for  these  model-data  comparisons.  Discussions  have  taken  plaee  with  NAVO, 
they  are  onboard  with  this  scoreeard  system  and  we  will  be  working  eollaboratively  with  them  on  its 
continued  development  and  implementation  in  FY16.  Issues  we’re  diseussing  with  NAVO  include 
implementation  of  new  data  streams  and  refining  the  constituents  of  the  seorecards  to  meet  their 
operational  needs.  We  are  planning  a  technieal  report  in  FY16  detailing  this  scorecard  package, 
examples  of  metries  and  real-world  examples  of  its  value  in  revealing  system  performance  issues. 

IMPACT/APPLICATIONS 

The  metries  and  diagnostics  developed  in  this  projeet  will  help  to  evaluate  and  result  in  the  ability  to 
routinely  monitor  the  coupled  model  system  assisting  in  the  development  of  a  skillful  eoupled 
ensemble  foreeast  system  for  intraseasonal  and  seasonal  time  seales. 
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TRANSITIONS 


Some  the  tools  developed  under  this  project  were  transitioned  to  WU-8.  The  score  card  that  will  be 
proposed  in  FY16  will  be  transitioned  to  other  units  of  the  ESPC  project. 

RELATED  PROJECTS 

This  project  is  related  to  other  units  of  the  ESPC  project  through  gathering  of  the  diagnostics 
developed  under  other  units  (e.g.  ice  diagnostics,  process  oriented  atmospheric  diagnostics)  as  well  as 
6.2  MJO  base  effort  and  ONR  Seamless  Prediction  effort.  The  diagnostics  development  will  depend  on 
availability  of  hindcasts  and  reanalysis  of  the  ESPC  coupled  system.  The  feedback  from  other  units 
will  impact  the  refining  of  the  score  card  and  choice  of  process  oriented  diagnostics. 
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